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Abstract. We recover Gessel's determinantal formula for the generating function of per- 
mutations with no ascending subsequence of length m + 1. The starting point of our proof 
is the recursive construction of these permutations by insertion of the largest entry. This 
construction is of course extremely simple. The cost of this simplicity is that we need to take 
into account in the enumeration m — 1 additional parameters — namely, the positions of the 
leftmost increasing subsequences of length i, for i = 2, . . . , m. This yields for the generating 
function a functional equation with m — 1 "catalytic" variables, and the heart of the paper is 
the solution of this equation. 

We perform a similar task for involutions with no descending subsequence of length m + 1, 
constructed recursively by adding a cycle containing the largest entry. We refine this result 
by keeping track of the number of fixed points. 

In passing, we prove that the ordinary generating functions of these families of permutations 
can be expressed as constant terms of rational series. 



1. Introduction 

Let t = r(l) • • • r(n) be a permutation in the symmetric group ©„. We denote by |r| := n 
the length of r. An ascending (resp. descending) subsequence of r of length k is a fc-tuple 
(r(ii), . . . ,r(«fc)) such that i\ < ■ ■ • < and t(ii) < • • ■ < rii^) (resp. > • • ■ > r(ifc)). For 

m > 1, the set of permutations in which all ascending subsequences have length at most m is 
denoted by &( m \ In pattern-avoidance terms, the permutations of ©( m ) are those that avoid the 
increasing pattern 12 ■ • • m{m + 1), and an ascending subsequence of length k is an occurrence 
of the pattern 12 • • • k. The set of 12 • • • m(m + l)-avoiding permutations of length n is denoted 
©l™"*. Note that several families of pattern avoiding permutations are equinumerous with &^ 
(see [2, 26]). 

In 1990, Gessel proved a beautiful determinantal formula for what could be called the Bessel 
generating function of permutations of & m '. This formula was the starting point of Baik, Deift 
and Johansson's study of the distribution of the longest ascending subsequence in a random 
permutation [3]. 

Theorem 1 ([16]). The Bessel generating function of permutations avoiding 12 ■■■ m(m + 1) is 

LU|2 = d0t (^-->)l<-t,j<m ' 

where 

h= ^ n\(n + i)\- (1) 

n>max(0,-i) 



Note that Ii = I-i, and that we can more loosely write 



^-^ n\(n + i)\ ^— ' n\(n — i)\ ' 

n>0 V ' n>0 V ; 
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provided we interpret factorials as Gamma functions (in particular. = 1/T(i + 1) = if 
i < 0). 

Gessel's original proof was algebraic in nature [16]. He first established a determinant al 
identity dealing with Schur functions (and hence with semi-standard Young tableaux, whereas 
the above theorem deals, via Schensted's correspondence, with standard tableaux). He then 
applied to this identity an operator 9 that extracts certain coefficients, and this led to Theorem 1. 
A few years later, Krattenthaler found a bijective proof of Gessel's Schur function identity [23], 
which specializes into a bijective proof of Theorem 1. Then, Gessel, Weinstein and Wilf gave 
two bijective proofs of this theorem, involving sign-reversing involutions [17]. Two other proofs, 
involving Young tableaux, were recently published by Novak [30] and Xin [42] . 

For small values of to, more proofs of Theorem 1 have been given. In particular, there exists a 
wealth of ways of proving that the number of 123-avoiding permutations of ©„ is the rfi 1 Catalan 
number ( 2 ^) /(n + 1), and numerous refinements of this result [7, 8, 14, 22, 25, 34, 35, 36, 40]. The 
laziest proof (combinatorially speaking) is based on the following observation: a permutation 7r 

(2) .... . . (2) 

of ©„_j_x is obtained by inserting n+1 in a permutation r of . To avoid the creation of an 
ascending subsequence of length 3, the insertion must not take place to the right of the leftmost 
ascent of r. Hence, in order to exploit this simple recursive description of permutations of &( 2 \ 
one must keep track of the position of the first ascent. Let us denote 

, , _ J n + 1, if r avoids 12; 

^ ' 1 min{z : r(i — 1) < t(£)}, otherwise, 

and define the bivariate generating function 

F(u;t) := «° CT)_1 t |r| . 
ree< 2 > 

It is not hard to see (and this will be explained in details in Section 2) that the recursive 
description of permutations of S*- 2 -* translates into the following equation: 

1-t -)F(u;t) = l-t -F(l;t). (2) 

v u — 1 J u — 1 

The variable u is said to be catalytic for this equation. This means that one cannot simply set 
u = 1 to solve for F(l;t) first. However, this equation can be solved using the so-called kernel 
method (see, e.g., [5, 11, 33]): one specializes u to the unique power series U that cancels the 
kernel of the equation (that is, the coefficient of F(u;t)): 



2t 

This choice cancels the left-hand side of the equation, and thus its right-hand side, yielding the 
(ordinary) length generating function of 123-avoiding permutations: 



v ' ; tU 2t ^ n + 1 

n>0 

It is natural to ask whether this approach can be generalized to a generic value of to: after 
all, a permutation 7r of is still obtained by inserting n+1 in a permutation r of 6^"^. 

However, to avoid creating an ascending subsequence of length to + 1, the insertion must not take 
place to the right of the leftmost ascending subsequence of length to of r. In order to keep track 
recursively of the position of this subsequence, one must also keep track of the position of the 
leftmost ascending subsequence of length m — 1. And so on! Hence this recursive construction 
(often called the generating tree construction [40, 41]) translates into a functional equation 
involving to — 1 catalytic variables U2, ■ ■ ■ , u m . The whole point is to solve this equation, and 
this is what we do in this paper. Our method combines three ingredients: an appropriate change 
of variables, followed by what is essentially the reflection principle [18], but performed at the level 
of power series, and finally a coefficient extraction. To warm up, we illustrate these ingredients 
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in Section 3 by two simple examples: we first give another solution of the equation (2) obtained 
when to = 2, and then a generating function proof of MacMahon's formula for the number of 
standard tableaux of a given shape. 

What is the interest of this exercise? Firstly, we believe it answers a natural question: we 
have in one hand a simple recursive construction of certain permutations, in the other hand a 
nice expression for their generating function, and it would be frustrating not to be able to derive 
the expression from the construction. Secondly, the combinatorial literature abunds in objects 
that can be described recursively by keeping track of an arbitrary (but bounded) number of 
additional (or: catalytic) parameters: permutations of course, but also lattice paths, tableaux, 
matchings, plane partitions, set partitions... Some, but not all, can be solved by the reflection 
principle, and we hope that this first solution of an equation with to catalytic variables will be 
followed by others. 

In fact, we provide in this paper another application of our approach, still in the field of 
permutations: We recover a determinantal formula for the enumeration of involutions with no 
long descending subsequence [16]. Let (resp. ul™' 1 ) denote the set of involutions (resp. 

involutions of length n) avoiding the decreasing pattern (to + l)m ■ ■ ■ 21. Again, several families 
of pattern avoiding involutions are equinumerous with ©1"^ (see [12, 13, 21, 26]). 

Theorem 2. The exponential generating function of involutions avoiding (to + 1)to ■ • • 21 is 

tM f e* det - Ii+j)^^ ' ifm = 2t+l- 



E 



, |T| 



|! \ det (Ii-j +i»+i-i) 1 < iji < <) ifm = 2£, 
where Ii is defined by (1). 



This result is obtained by applying Gessel's 9 operator to a Schur function identity due to 
Bender and Knuth [6] . The latter identity has been refined by taking into account the number of 
columns of odd size in the tableaux (see Goulden [19]; Krattenthaler then gave a bijective proof of 
this refinement [23]). Using the operator 9, and the properties of Schensted's correspondence [38, 
Exercise 7.28], this translates into a refinement of Theorem 2 that takes into account the number 
/(t) of fixed points in r. We shall also recover this result. 

Theorem 3. If to = 11 + 1, the exponential generating function of involutions avoiding (m + 
1)to • • ■ 21 and having p fixed points is 

]3y - r[ det (ti-j ~ I i+j)i<i,j<e ■ 
rea< m ),/(r)= P 1 

If m = 2£, this generating function is 

* |t| = dct ( ( 1 p+(-j - I P +e+j)i<j<( 
,, \t\\ V C^M-j-i — Ii-j-i)2<i<e,i<j<e , 

fM=p 

where we have described separately the first row of the determinant and the next £ — 1 rows 
(i = 2,...,£). 

The first result of Theorem 3 can be restated as follows: if m = 2£+l, the generating function 
of involutions avoiding (m + l)m • ■ ■ 21, counted by the length and number of fixed points is 

E ^ s/(T) = eS * dct (^-^) 1 < lJ <,- (3) 

It thus appears as a very simple extension of the first part of Theorem 2, and indeed, the 
connection between these two formulas is easy to justify combinatorially (the fixed points play 
no role when one forbids a decreasing pattern of even length). 
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Let us now outline the structure of the paper. In Section 2, we describe how the "catalytic" 
parameters change in the recursive construction of permutations of &^ m ^ and 3^ m \ We do not 
give the proofs, as this was done by Guibert and Jaggard & Marincel, respectively [20, 21]. We 
then convert these descriptions into the functional equations that are at the heart of this paper 
(Propositions 5 and 7). In Section 3, we illustrate our approach by two simple examples, namely 
the enumeration of permutations of @^ and of standard Young tableaux. Next we return to 
permutations: we first address in Section 4 the solution of the equation obtained for involutions 
of 3K m ), and finally, we solve in Section 5 the equation obtained for permutations of &( m K The 
reason why we address involutions first is that the solution is really elementary in this case. One 
step of the solution turns out to be more difficult in the case of permutations, although the basic 
ingredients are the same. 

Let us finish with some standard definitions and notation. Let A be a commutative ring and 
x an indeterminate. We denote by A[x] (resp. .A [[a;]]) the ring of polynomials (resp. formal 
power series) in x with coefficients in A. If A is a field, then A(x) denotes the field of rational 
functions in x (with coefficients in A). This notation is generalized to polynomials, fractions 
and series in several indeterminates. We denote x = 1/x, so that A[x, x] is the ring of Laurent 
polynomials in x with coefficients in A. A Laurent series is a series of the form J2 n >n a(n)x n , 
for some no G Z. The coefficient of x n in F(x) is denoted \x n ]F(x). 

Most of the series that we use in this paper are power series in t with coefficients in 
that is, series of the form 

F(x;t)= J2 f(*'^)xH n , 

ri>0,iSZ 

where for all n, almost all coefficients f(i; n) are zero. The positive part of F(x; t) in x is the 
following series, which has coefficients in xA[x]: 

[x>]F(x;t) := £ f(i;n)xH n . 

n>CM>0 

We define similarly the negative, non-negative and non-positive parts of F(x; t) in x, which we 
denote respectively by [x < }F(x; t), [x-]F(x;t) and [x-]F(x; t). 

2. Generating trees and functional equations 
2.1. Permutations avoiding 12 • • • m(m + 1) 

Take a permutation ir of S„"l 1; written as the word 7r(l) • ■ ■ n(n + 1). Erase from this word 

the value n + 1: this gives a permutation r of &n . This property allows us to display the 
permutations of £>('") as the nodes of a generating tree. At the root of this tree sits the unique 
permutation of length 0, and the children of a node indexed by r G &n are the permutations 
of S^™ i obtained by inserting the value n + 1 in r. In how many ways is this insertion possible? 
If r avoids 12 • • • m, then all insertion positions are admissible, that is, give a permutation of 
©i'+i- There are n + 1 such positions. Otherwise, only the a leftmost insertion positions are 
admissible, where a is the position of the leftmost occurrence of 12 ■ • • m in r. More precisely: 

a = min{i m : 3 i\ < i 2 < ■ ■ ■ < i m s.t. r(ii) < • • • < r(i m )}. 

As we wish to describe recursively the shape of the generating tree, we now need to find the 
position of the leftmost occurrence of 12 • • • m in the children of r. But it is easily seen that this 
depends on the position of the leftmost occurrence of 12 • • • (m — 1) in r. And so on! We are 
thus led to define the following m parameters: for 1 < j ' < m, and r G &n , let 

, , J n+1, if r avoids 12 ••• j; , . 

1 mm{ij : 3 ii < i 2 < ■ ■ ■ < ij s.t. r(ii) < • • • < r(ij)}, otherwise. 

Note that ai(r) = 1, and that ai(r) < • ■ ■ < a m (r). We call the sequence L(t) := (a2(r), . . . , a m (r)) 
the label of r. The empty permutation has label (1, . . . , 1). 
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We can now describe the labels of the children of r in terms of L(r) (Guibert [20, Prop. 4.47]). 

Proposition 4. Let r G &n with L(t) = (a 2 , . . . , a m ). Denote a\ = 1. TTie labels of the a m 
permutations of obtained by inserting n + 1 in t are 

f (a 2 + l,o 3 + 1,. . .,o m + 1) 

1 (a 2 , . . . , Oj_i, a, a_j +1 + 1, . . . , a m + 1) for 2 < j < m and aj-% + 1 < a < dj. 

The first label corresponds to an insertion in position 1, while the label involving a corresponds 
to an insertion in position a. We refer the reader to Figure 1 for an example. 
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FIGURE 1. The permutation r = 859613742 G 6^ j . One has oi(r) = 1, 
a 2( T ) = 3, 03 (r) = 7. There are 7 admissible ways to insert the value 10. 
Inserting 10 to the right of r(7) would create an occurrence of 1234. 



Let us now translate the recursive construction of permutations of ©( m ' in terms of generating 
functions. Let F(u 2 , ■ ■ ■ ,u m ;t) be the (ordinary) generating function of permutations of &( m \ 
counted by the statistics 02, ... , a m and by the length: 

F(u 2 ,...,u m ;t) = ]T u 2 ° 2(T) ---<7^H 



V>o • • • \l™ 



where F 02 ,...,a TO (t) is the length generating function of permutations of 0( m ) having label (a 2 , . . . , a rn ). 
We still denote 01 = 1. The above proposition gives 



F(u 2 , • ■ • , u m \ t) = u 2 ■ ■ ■ u m + tu 2 ■ ■ ■ u m F(u 2 , u m ; t) 

m a 

■ U 

a,2,...,a, m j-2 a-dj-i+l 

Using 

a 4 a 7 + 1 i — 1 + 1 

It/ — Uj 



we obtain (given that eti = 1): 

F(u; i) = M 2 , m + tu 2 , m F(u; t) + tu 2 , m — 



u 2 — 1 

U-i — l 

where F(u; t) = F{u 2 , ■ ■ ■ , u m ;t) and u^fc = vtjUj+i ■ ■ -Uk- 



^(u; t) - F(tt 2 , ■ ■ ■ , Mj-2, Uj-lUj, L Mj+li ■ ■ ■ 1 "mi t) 
3=3 J 
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To finish, let us perform an elementary transformation on the series F(u; t). Define 

F(v;t)=F( Vl ,...,v m ;t)= £ v^v^ ■ ■ ■ t ,M+ 1 -»™fM, (6) 

ree< m > 

where (02,..., a m ) = L(t). We have eliminated the dependence a 2 < ••• < a m between the 
exponents of 112, ■ ■ . , u m in i). As will be seen below, another effect of this change of series 
is that the cases j = 2 and j = 3, . . . , m now play the same role. We also note that the variable 
t is now redundant in F(v; i), but it is our main variable, and we find it convenient to keep it. 
The series F and F are related by 

ev j.\ v m B ( «l «m-l 

Vl \v 2 v m 

and conversely 

F(u 2 , ■ ■ ■ F(u2, m v m> u 3>m v m , u m v m , v m ; t) 

where as above Uj t k = u j u j+i 1 ■ ■ u k- The functional equation (5) satisfied by F(u;t) translates 
into an equation of a slightly simpler form satisfied by F(v;t). 

Proposition 5. The generating function F(v;t) = F(vi, . . . ,v m ;t) of permutations of &( m \ 
defined by (6), satisfies 



F(v;t) = I + tviF(v;t) + t^Vj-xVj 



J=2 



F(v;t) - Fjvi, . . .,Vj- 2 ,Vj,Vj,Vj+i, ...,v m ;t) 

Vj-l- Vj 



The series F(l, . . . , 1; t) counts permutations of ©( m ) by their length. 

In Section 5, we derive from this equation the Bessel generating function of permutations of 
<S^ m \ as given by Theorem 1. 

2.2. Involutions avoiding (to + 1)to • • • 21 

It follows from the properties of Schensted's correspondence [37] that the number of involutions 
of length n avoiding 12- ■■m(m + 1) equals the number of involutions of length n avoiding 
(to+1)to • • • 21. However, this correspondence is not a simple symmetry, and the generating trees 
that describe 12 • • • m(m + l)-avoiding involutions and (to + l)m • • • 21-avoiding involutions are 
not isomorphic. Both trees are defined by the same principle: the root is the empty permutation 
and the parent of an involution ir is obtained by deleting the cycle containing the largest entry, 
and normalizing the resulting sequence. For instance, if n = 426153, the deletion of the 2-cycle 
(3, 6) first gives 4215, and, after normalization, 3214. 

The tree that generates 12 • • • to (to + l)-avoiding involutions is similar to the tree generating 
12 ■ ■ • to(to + l)-avoiding permutations. Its description involves to catalytic parameters (Gui- 
bert [20, Prop. 4.52]). The tree that generates (to + 1)to ■•• 21-avoiding involutions requires 
[to/2J catalytic parameters only (Jaggard & Marincel [21]). The source of this compactness 
is easy to understand: an involution t contains the pattern k ■ ■ ■ 21 if and only if it contains a 
symmetric occurrence of this pattern (by this, we mean that the corresponding set of points in 
the diagram of r is symmetric with respect to the first diagonal, see Figure 2). Equivalently, this 
means that a decreasing subsequence of length \k/2] occurs in the points of the diagram lying 
on or above the first diagonal. Thus we only need to keep track of descending subsequences of 
length at most to/2 (in the top part of the diagram), and we can expect to have about to/2 
catalytic parameters. 

Let us now describe in details the tree generating (to + 1)to • ■ ■ 21-avoiding involutions. The 
example of Figure 2 illustrates the argument. Let r be an involution of 2f« . Inserting n + 1 
as a fixed point in r always gives an involution of J^K For 1 < i < n + 1, let us now consider 
the permutation tt obtained by adding 1 to all values larger than or equal to i, and inserting 
the 2-cycle (i,n + 2). How many of these insertions are admissible, that is, give an involution 



PERMUTATIONS WITH NO LONG MONOTONE SUBSEQUENCE 



^ ^ ^ ^ 




FIGURE 2. The involution r = 3 2 1 12 7 9 5 8 6 11 10 4 e 3$. 0ne has 
ai(r) = 3, o,2(t) ~ 9. There are 9 admissible ways to insert a 2-cycle. 



of 3K m )? If t avoids (m — 1) • • • 21, then all insertions are admissible, including the most "risky" 
one, corresponding to i = 1. Otherwise, the only admissible values of i are n+ 1, n, . . . , n — a + 2, 
where n — a + 1 is the position of the rightmost symmetric occurrence of (m — 1) • • • 21. In other 
words, if we denote m = 2£ + e with e € {0, 1}, 

n — a + 1 = maxjii : 3 ix < i% < ■ ■ ■ < it s.t. r(ii) > • • • > r(ig) > ie + e}. 

Again, in order to keep track of this parameter recursively, we are led to define, for 1 < j < £, 
the following I catalytic parameters: 



aj(r) 



n+1, if t avoids (2j - 1 + e) ■■• 21; 

n + 1 — maxjii : 3 ii < 12 < • ■ ■ < ij s.t. r(ix) > ■ ■ • > T(ij) > ij + e}, otherwise. 



In particular, a^(r) is the parameter that was denoted a above, and it is also the number of 
admissible insertions of a 2-cycle in r. We call the sequence L(t) := (ai(r), . . . , an (r)) the label 
of t. Note that ai(r) < ■ ■ • < ai(r). The empty permutation has label (1, . . . , 1). 
We can now describe the labels of the children of r in terms of L(t). 

Proposition 6 (Jaggard & Marincel [21]). Let r be an involution in 3^ with L{t) = (oi, . . . , ai). 
Denote ao = 0. The labels of the ae involutions of 3^ obtained by inserting a cycle in r are 



(a\ + l,a2 + 1, • • -,ae + 1), 

(l,a2 + 1, • • • ,ae + 1), 

(oi + 1, . . . , a,j-i + 1, a, Oj+i + 2, 



if m is odd; 
if m is even; 
+ 2) forl<j<£ and Oj_i + 2 < a < aj + 1. 



The first two labels correspond to the insertion of a fixed point, the other ones to the insertion 
of a 2-cycle. 

We refer again the reader to Figure 2 for an example. 

Let us now translate the recursive construction of involutions of in terms of generating 
functions. Let G(ui, . . . , uf, t) be the (ordinary) generating function of involutions of 3^ m \ 
counted by the statistics a\ , . . . , ae and by the length: 



G(ui,...,U£]t) = 



ai(r) 



C {T) t^ 



G ai ,...Mi(t)Ui 1 
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where G ai ,...,a t {t) is the length generating function of permutations ofO^" 1 -* having label (ai, . . . ,ai). 
We still denote ao = 0. The above proposition gives 

G(ui, ...,Uf,t) =Ux---ui + tUf- u t G(ui, . . . , ug ; t)x m =i + tu x - ■■ u t G(l, u 2 ,...,u t ; t)x m =o 



t 2 Yl G au ..., ae (t)J2 E < i+1 ---^ 1+1 ^iV +2 ---u^ +2 



ai,...,a^ j — 1 a— aj_i+2 

where Xm=i equals 1 if m equals i modulo 2, and otherwise. Using 

Gu+l a 7 +2 0,7-1+2 



a— aj_i+2 



Uj - 1 



we finally obtain (given that ao = 0): 

G(u; t) = ui t e + tu lie G(u; t)x m =l + tui^G{l, u 2 ,...,u e ; t)x m =o 

,,2 G(u;i) - G(ui, . . . ,Uj-2,Uj-iUj, 1, u j+1 , . . . ,uf,t) 

j=l U 3 

where G(u; t) = G{u\ 1 . . . , ug; t) and Uj t k = u jUj+i ■ ■ ■ Uk- 

To finish, let us perform an elementary transformation on the series G(u;t). Define 

G(v;t) = G(v 1 ,...,v e ;t)= ]T v?v?- a i---v a /- ae -H^\, (8) 

Te3 ( m ) 

where {ax,...,ai) = £{t). We have eliminated the dependence a\ < ■ ■ ■ < an between the 
exponents of ui,...,ut in G(u;t). The series G and G are related by 

G(vi, ...,v t ;t) = G ( -^,vf,t) , 

\v 2 v e J 

and conversely 

G(ui,...,u e ;t) = G(ui t e,U2,e,---,ut,t) 

where as above Uj t k = u j u j+i 1 ■ ■ u k- The functional equation (7) satisfied by G(u;t) translates 
as follows. 

Proposition 7. The generating function G(v;t) = G{v\, ...,Vf,t) of involutions of3^ m ', defined 
by (8), satisfies 

G(v; t) =v± + tviG{v; t)x m =i + tviG(v 2 , v 2 , v 3 , ...,v e ; t)x m =o 

G(v;t) - G(v±, . . .,Vj-x,v j+ x,v j+ x,v j+ 2, ...,v e ;t) 



Vj - v j+ i 



The series G(l, . . . , l;i) counts involutions of DK m ' 6y i/ieir length. 

In Section 4, we derive from this equation the exponential generating function of involutions 
of j( m \ as given by Theorem 2. We then refine the result to take into account the number of 
fixed points. 



3. TWO EXAMPLES 

In this section, we illustrate the ingredients of our solution of the equations of Propositions 5 
and 7 by taking two examples. The first one deals with the enumeration of 123-avoiding permu- 
tations. The second one is a generating function proof of MacMahon's formula for the number 
of standard tableaux of a given shape, and should clarify what we meant in the introduction by 
"the reflection principle performed at the level of power series". 
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3.1. Permutations avoiding 123 

In the introduction, we wrote the following equation for the bivariate generating function of 
123-avoiding permutations, counted by the position of the first ascent and the length: 

u \ u 

1 - t F(u; t) = l-t -F(l; t). 

u — 1 J u — 1 

This is the case m = 2 of Proposition 5, with v\ = u and v-i = 1. 

As explained in Section 1, this equation can be solved by an appropriate choice of u that 
cancels the kernel, and thus eliminates the unknown series F(u;t). This is the standard kernel 
method. We present here an alternative solution, sometimes called the algebraic kernel method [9, 
10], where instead F(l;t) is eliminated. This elimination is obtained by exploiting a certain 
symmetry of the kernel. This symmetry appears clearly if we set u = 1 + x. The equation then 
reads: 

(1 - t (1 + x)(l + x)) F(l + x; t) = 1 - t(l + x)F(l; t) 
with x — 1/x. The kernel is now invariant under x <— > x. Replace x by x: 

(1 - t (1 + x){l + x)) F(l + x;t) = 1 - t(l + x)F(l; t). 
We now eliminate F(l;t) by taking a linear combination of these two equations. This leaves: 

(l-t(l + x)(l+x))(F(l + x;t)-xF(l + x;t)) = 1 - x, (9) 

or 

F (i + x; t) - mi +*;*) = r^r^TTi) := R(x > t] - 

In this equation, 

- F(l + x; t) is a series in t with coefficients in Q[x], 

- xF(l + x;t) is a series in t with coefficients in xQ[a;], 

- the right-hand side R(x;t) is a series in t with coefficients in Q[x,x]. 

Consequently, F(l + x;t) is the non-negative part of R(x;t) in x. In particular, the length 
generating function of 123-avoiding permutations is 

F(l;t) = [x°]R(x;t) = ^[x°](l - x).t"(1 + .x) 2n t n (10) 

n>0 

- £ff 2 ;W„ 2 ;,)> 



y — 

n + 1 



n J \n + 1 
i" (In 



V n 



n>0 

This small example contains all ingredients of what will be our solution for a generic value of m: 

- a change of variables, which may not have a clear combinatorial meaning, 

- a finite group G acting on power series that leaves the kernel unchanged (here, the group 
has order 2, and replaces x by 1/x), 

- a linear combination (9) of all the equations obtained by letting an element of G act 
on the original functional equation; in this linear combination, called the orbit sum, the 
left-hand side is a multiple of the kernel, and the right-hand side does not contain any 
unknown series, 

- finally, a coefficient extraction (10) that gives the generating function under interest. 

Let us mention, however, that for a generic value of m, the change of variables used in Section 5 
is not a direct extension of v n- 1 + x. But, on this small example, the latter choice is simpler. 
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3.2. Standard Young tableaux 

Let A = (Ax, • ■ • , A m ) £ N m be an integer partition. That is, Ax > • • ■ > A TO > 0. The weight 
of A is |A| := Ax + • ■ ■ + A m . We identify A with its Ferrers shape, in which the i^ 1 row has A; 
cells. A standard tableau of shape A is a filling of the cells of A with the integers 1,2,..., |A|, 
that increases along rows and columns (Figure 3). The height of the tableau is the number of 
non-empty rows, that is max(i : A» > 0). Let f x denote the number of standard Young tableaux 
of shape A. 



1 


2 


4 


8 


3 


5 


9 




6 


7 


10 



Figure 3. The Ferrers shape associated with the partition A = (4,3,3) and a 
standard tableau of shape A. 



Our objective here is to recover the hook-length formula, or, rather, an equivalent form due 
to MacMahon [29, Sec. Ill, Chap. V]. 

Proposition 8. Let A = (Ax, . . . , A m ) be a partition of weight n. The number of standard Young 
tableaux of shape A is 

/A= n m 1 (A i -» + r»)! II (Ai-A,-i + i). 

Proof. Let F(u) = F(u±, . . . , u m ) be the generating function of standard tableaux of height at 
most to: 

m 

Ai>— >A m >0 i=l 

For j — 2, . . . , to, we denote by Fj{u\, . . . , %-2, Uj—iUj, u j+ii ■ ■ ■ , u m) = Fj{ u ) the generating 
function of standard tableaux such that the parts Aj_x and Xj are equal. This series is obtained 
by extracting the corresponding terms from F(u) (it is also called the (J — 1 , j')-diagonal of F(u)). 
In all terms of this series, Uj-i and Uj appear with the same exponent, which allows us to write 
this series in the above form. 

Now a tableau of weight n + 1 is obtained by adding a cell labelled n + 1 to a tableau of 
weight n. This cell can be added to the j^ 1 row unless this row should have the same length as 
the (J — l) st row. This gives directly the following equation: 

m 

F(u) = 1 + Ul F(u) + E «j ~ Fj («)) . 

that is, 

m m 

( i -E«i) jr («) = i -£ u ^( u )- 

Observe that the kernel K{u) := 1 — i s invariant under the action of the symmetric group 
© m , seen as a group of transformations of polynomials in u\, . . . , u m . This group is generated 
by to — 1 elements of order 2, denoted ax, ... , o~ m -\\ 

CTj(P(u-i,. . . , U m )) = P(U1, • ■ • , Uj-l,Uj + i,Uj,Uj + 2, ■ ■ ■ ,U m ). 

Let us multiply the equation by M(u) := u" 1 " 1 ■ ■ ■ This gives: 

rn 

k(u)m(u)f(u) = m(u) Yi < _1 • • • «ri tf_1)u r j ' +1 • • • < f ^- w 

J=2 
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Recall that Fj(u) stands for Fj{u\, . . . , Uj— 2, Uj—iUj, %+i, • ■ • , u m ). Hence the j^ 1 term in the 
above sum is invariant under the action of the generator <7j-i (which exchanges Uj-i and Uj). 
Consequently, forming the signed sum of (11) over the symmetric group © m gives the following 
orbit sum, which does not involve the series Fj : 

J2 a(K(u)M(u)F(u)) = ]T e(a) a(M(u)) , 

<T6S m CreSm 

or, given that K(u) is @ m -invariant, 

2^ e(<r) (t(M(w)F(m)) = " ^, , • (12) 

Of course, the sum on the right-hand side can be evaluated explicitly (the numerator is the 
Vandermonde determinant), but this will not be needed here. 

We claim that the number f x can be simply obtained by a coefficient extraction in the 
above identity. Consider the series M(u)F(u). Each monomial uj 1 • • • uf™ that occurs in it 
satisfies a\ > ■ ■ ■ > a rn (because cij = m — i + Aj, where A is a partition). Consequently, if a 
is not the identity, the exponents of any monomial aj 1 • • • occurring in a (M(u)F(u)) are 
totally ordered in a different way. Hence, when we extract the coefficient of «™~ 1+Al ■ • • u^~ Xm 
from (12), only the term corresponding to a = id contributes in the left-hand side, so that 



Eu66 m S i cr ) °-(M(u)) 



K{u) 



Given that M(u) = u" 1 1 • • • u 1 m _ 1 u®. n and 

1 1 >^ (£»! + ■ 



E 



k(u) i-e-i«7 ^ n, 

V ) S-<] = \ J oi,...,o TO >0 1U 

we obtain 

(Ai + ••■ + A m )! 



i x = E 



n\ det 
n! det 



n™ 1 (A l - J+( r-i( J ))! 

1 



(A,-z+j)7i< 4J < m 
(Aj-x + j + l)---(A. t -z + m) 
(A, — i + m)\ 



l<z,_7<m 



UT=i (A.- . + rn)! 



det ((Aj - i + j + 1) • • • (A; - i + m)) 1 < 4J -< m . 



The (i, j)-coefficient of the latter determinant is a polynomial in Xi — i of degree m— j and leading 
coefficient 1. Hence the determinant is simply the Vandermonde determinant det((Aj — i) m ~ J ), 
that is, rii<j(^i — A? ~ * + .?)• This completes the proof of the proposition. B 

We recognize in this proof three of the four ingredients that were used in the enumeration 
of 123-avoiding permutations: the finite group that leaves the kernel invariant (here, S m ), the 
orbit sum (12), and the final coefficient extraction. In this example, the symmetries of the kernel 
are obvious already with the original variables Ui, so that no change of variables is required. 

This proof is the generating function counterpart of the classical proof that encodes tableaux 
of height at most m by paths in N m formed of unit positive steps, that start from (0, . . . , 0) and 
remain in the wedge x\ > ■ ■ ■ > x m > 0, and then uses the reflection principle. It is also very 
close to another proof due to Xin [42, Sec. 3.1]. 
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4. Involutions with no long descending subsequence 

We now address the solution of the functional equation of Proposition 7, which defines the 
generating function of involutions avoiding (m + l)m ■ ■ ■ 21. 

4.1. IN VARIANCE PROPERTIES OF THE KERNEL 

As discussed in the previous section, our objective is to exploit invariance properties of the 
kernel, that is, the coefficient of G(v;t). Let us first divide the equation of Proposition 7 by v\. 
Then the kernel reads 



1 2 VjVj+1 
tXm=l —t > • 



<Jj{P(xi, . . .,Xi)) 



,xg) 



The invariance properties of this rational function appear clearly after performing the following 
change of variables: 

Vi = ; r. (13) 

Indeed, the kernel becomes 

K(x; t) = 1 - t(xi H h xi) - txm=i - t(xi H h x£), 

where Xi = 1/xi, and is invariant under the action of the hyperoctahedral group Bg (the group of 
signed permutations), seen as a group of transformations on Laurent polynomials in x\, . . . , X£. 
This group is generated by £ elements of order 2, denoted cr\,...,af. 

P(x 1 ,x 2 ,.-.,x l ), if .7 = 1; 

P(xi, . . . ,Xj-2,Xj,Xj-i,x j+ i, . . .,xt), for j > 2. 

The equation of Proposition 7 now reads: 

e 

K(x;t)G(x;t) = 1 + iG(0, x 2 , ■ ■ ■ ,xt)x m =o - t/,XjG(xi, . . . ,Xj-2,x 3 -i + x^O^xj+i, . . . 

3=1 

where 

G(x; t) = G(x\, . . . , xi;t) = G ( r, -, . . . , ; t 

\ 1 — t(x\ + • • ■ + xg) 1 — t(X2 + • • • + xg ) 1 — txg 

4.2. Orbit sum 

We now handle separately the odd and even case. 
• If m is odd, the equation reads 

i 

K(x; t)G(x; t) = 1 — t XjG(x±, . . . , Xj-2, Xj-i + %j, 0, ajj+i, ■ • ■ , xg; t) 
j=i 

where 

K (x; t) = 1 - i(l + x\ H h x e + xi H \-xg). (14) 

Let us multiply the equation by 

M(x) := xxxj ---Xg. (15) 

This gives: 

K(x]t)M(x)G(x;t) = M{x) 

e 

- ty]xi ■ ■ -x^ZfX^" x 3 ^ ■ ••x\G{x\,. ■ .,Xj-2,Xj-\ + Xj,0,x j+ i,. . .,xg;t). (16) 
i=i 

The first term (j = 1) of the sum reads x\ ■ ■ ■ x e gG(0, X2 ■ ■ ■ , xg) and is invariant under the action 
of the generator <j\ of Bg (which replaces x\ by xi). For j > 2, the j^ 1 term of the sum is 
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invariant under the action of the generator e>j (which exchanges Xj-i and Xj). Consequently, 
forming the signed sum of (16) over the hyperoctahedral group Bg gives the following orbit sum: 

e(a) a(K(x;t)M(x)G(x;t)) = £ s(a)a(M(x)), 

or, given that K(x\ t) is ^-invariant, 

£ e{a)a{M{x)G{x;t)) = (17) 

where K(x;t) is given by (14) and M(x) by (15). 

• If m is even, the equation reads 

l 

K(x;t)G(x;t) = l+t(l— xi)G(0, X2, ■■■,xt, t)—t' s ^ j XjG{xi, . . . , Xj-2,Xj-i+Xj,0,Xj + i, ...,xf,t) 
where 

K(x; t) = 1 - t(xi + hX£ + 5iH \-x~i), (18) 

Let us multiply the equation by 

M{x) :=x 2 xl---x i f 1 (l-x 1 )---(l-x e ). (19) 

This gives: 

e 

K(x;t)M(x)G(x;t) = M(x)+tx 2 x\ ■ ■■x e f l {l - x x ){l - x x ) Y[(l ~ x j )G(0,x 2 , ■■■,xz;t) 

I I 

- 1 JJ(! - Xj)^2x 2 ■ ■ ■x 3 j ZiX J j ~ a^- +1 • • ■x £ f 1 G(x 1 , . . . , x 3 - 2 , x 3 -i + Xj,0,x j+ i, . . .,xf,t). 

3=1 i=2 

(20) 

The term involving G(0, x 2 , ■ ■ ■ , xi) is invariant under the action of the generator a\ of Be. For 
.7 ^2, the jk* 1 term of the sum is invariant under the action of the generator <jj. Consequently, 
forming the signed sum of (20) over the hyperoctahedral group Bi yields the orbit sum (17), 
where now K(x;t) and M(x) are respectively given by (18) and (19). 

4.3. Extraction of G(l,...,l;i) 

• Assume m is odd, and consider the orbit sum (17). For every a £ Bp, the term 

a(M(x)G{x; i)) = a [x x x\ ■■■x\G ( — -J — — , — -J — — , . . . , ; t 

\ \ 1 - t(a;i H 1- xi) 1 - i(.x 2 H V xt) 1 - txe 

is a power series in i with coefficients in Q[a;i, . . . , xe, x\, . . . , xe}- We will prove that the coefficient 
of x\---x\ in (17) reduces to (5(0, . . . , 0; t) = G(l, . . . , 1; i), which is the (ordinary) length 
generating function of involutions avoiding (m + l)m • • • 21. 

First, if a has some signed elements, all monomials in the x^s occurring in a(M(x)G(x;t)) 
have at least one negative exponent. Hence a(M (x)G(x; t)) does not contribute to the coefficient 
of X\ ■ ■ ■ x\. 

If a is not signed, it is a mere permutation of the Xj's. Each monomial occurring in cr(M (x)G(x; t)) 
is of the form x^ 1 ■ ■ ■ x e g , where the e^'s are positive. However, monomials with e\ = 1 only occur 
if cr(l) = 1 (because of the factor M(x) = x\x\ ■ ■ ■ x £ e ). But then, if we also want e 2 = 2, the only 
permutations a that contribute are those that satisfy er(2) = 2. Iterating this observation, we 
see that the only permutation a that contributes to the coefficient of the identity. 

Moreover, its contribution is clearly G(0, . . . , 0; t) = G(l, . . . , 1; t). 

Let us state this as a proposition, in which we have also made explicit the right-hand side of 
the orbit sum. 
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Proposition 9. If m = 2£ + 1, the ordinary generating function of involutions avoiding (m 
l)m • • • 21 is the coefficient of x\x\ ■ ■ ■ x\ in a rational function: 

C lt)~ V i M -\x.x 2 x l \ dct(x) - x^^e 

Gm[t) - 2-,* - [XlX2 Xel l-t(l + Xl + --- + x e + x 1 + --- + x e y 

Equivalently, the exponential generating function of these involutions is 



G ™ W ■= E S = e * bisa ■ " ■ */] det ((xj - xj) 



l<ij'<£ 



Proof. We have just argued that G m (t) is the coefficient of Xix| • • -x| in the right-hand side 
of (17). There remains to prove that 

^2 £{o-)o-{x\ ■■■x[)= det(x} - x" j ) 1 <, l . J <i. 

This is easily proved if we consider that a first replaces some Xj's by their reciprocals, and then 
permutes the Xj's. More precisely, there is a bijection between Bg and Gi x Z|, sending a to 
(-7T, ei, . . . , e^), with 7r € &e and £ {— 1, 1}, such that 

<T(P(x 1 ,... t xt)) = n(P(x?,...,x?)) and e(a) = e '=- 1 >. (21) 

Thus 



£ £ (aV(xi---x|) = Yl £ W E 

o-eBf TiE&i ei,...,e/6{ — 1,1} 



tt{i:ej=-l}_ei ^,2e 2 _ let 
• i 7 r(l) J '7r(2) X 7r(^) 



= e £ M n (4w - 4« 

TTgSf i=l 

= det(xj - Xj)i<i^<e- 

This gives the expression of G m (t). We then convert it into an expression for the exponential 
generating function Gm (t) by observing that the ordinary generating function 1/(1 — at) = 
^2 n t n a n corresponds to the exponential generating function ^2 n t n a n /n\ = exp(at). ■ 

Remark. The determinant occurring in the proposition can be evaluated in closed form (see, 
e.g., [24, Lemma 2]): 

det(x} - x))i<ij<£ = (xi • • • xt)~ l JJ(x? - 1) ((xj - Xj-)(1 - x iX] )) 

i=l l<i<j<£ 

but this is not needed here. 

• Assume now that m = 2£ is even. The identity (17) still holds, with K(x;t) and M{x) given 
by (18) and (19). Based on the study of the odd case, it would be tempting to extract the 
coefficient of X2 • • • x^ _1 in this identity. However, this will not give G(0, . . . , 0; t), as both a = id 
and a = o\ (the generator of Bi that replaces x\ by x\) contribute to this coefficient. But we 
note that each term in the equation is a multiple of P{x) := J\i=i0- ~ x i)- Hence we will first 
divide by P(x). Let us study the action of a e Be on P(x), with a described as in (21). We 
have: 

*(P(x))=n((l-x?)...(l-x e /))=ir(p(x) J] (-a*) j = (-l)« i:e *=- 1 >P(x) [] x v(i) . 

\ i:ei= — 1 / i'.e.i— — 1 
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Hence, denoting e = (ei, . . . , eg), x e = {x\ x , . . . , x\ l ) and N(x) = X2 ■ ■ ■ x 1 ^ 1 , dividing (17) by 
P(x) gives 

E e(n)n(N(x e )G(x e ;t) JJ Si J = — ^— E e(n) tt [J 2«] . 

•ee, \ i:ei=-l / ^ ' ' \ »«( V i:e<=-l / / 

(22) 

Let us now extract from the left-hand side the coefficient of X2 • • • Xg 1 . The argument is sim- 
ilar to the odd case. If e 7^ (1, . . . , 1), each monomial occurring in N(x e ) G(x e ; t) Y[ t e =-1 ^« 
contains a negative exponent, and thus cannot contribute. Now for e = (1,...,1), the term 
7r (N(x) G(x; t)) only contributes if 7T = id, and then its contribution is G(0, . . . , 0; t), the length 
generating function of involutions avoiding (m + l)m ■ ■ ■ 21. We obtain the following counterpart 
of Proposition 9. 

Proposition 10. If m = 2£, the ordinary generating function of involutions avoiding (m + 
l)m ■ ■ • 21 is the coefficient of rational function: 



det(z} 1 + x})i<i,j<* 



1 - t(xi + ■ ■ ■ + xe + xi + ■ ■ ■ + xt) 



Equivalently, the exponential generating function of these involutions is 



<i,i<£ 



Proof. We have just argued that G m (t) is the coefficient of x\ - ■ -x\ 1 in the right-hand side 
of (22). There remains to evaluate the numerator in the right-hand side: 



E e W * f ^oo II = E e « ^ ( e ri 

\ i:ei=-l / ir£6 e \ee{-l,lY i=1 

E £(^(n(*r 1 +^)) 

rG6 f \i=l / 



(i-l)e i -Xe i =-i 



= dct(a;}- 1 +x)). 

This gives the expression of G m (t). Taking the corresponding exponential generating function 
gives Gm(t). a 

Remark. Again, the determinant occurring in the proposition can be evaluated in closed 
form [24, Eq. (2.6)], but this is not needed here. 



4.4. Determinantal expression of the series 

• Let us assume that m is odd, and return to Proposition 9. Taking the exponential generating 
function rather than the ordinary one makes the extraction of the coefficient of x\ ■ ■ ■ x\ an 
elementary task, as all variables xj decouple. The series p defined by (1) arise naturally from 
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We have: 



= e* ^ £(tt) JJ (/^(i) - 7 i+7r(i) ) 
u-eSf i=i 

= e* det (J<_j - Ji+j)^^ . 
We have thus recovered the first part of Theorem 2. 

• If m is even, we start from Proposition 10. Again, the variables Xj decouple in the exponential 
generating function: 

Gt\t) = J2 ^wflN" 1 ] ((*r w_1 

tt€©^ i=l 

= X! £ ( 7r ) II (^-7^(0 + ^i+7T(i)-l) 

= det + ^+ J -i) 1 < iji < £ • 
We have thus recovered the second part of Theorem 2. 

Remark. The determinantal expression of G$ implies that this series is D-finite, that is, 
satisfies a linear differential equation with polynomial coefficients. However, this follows as well 
from the constant term expressions of Propositions 9 and 10 using the closure properties of 
D-finite series [28, 27]. 

4.5. The number of fixed points 

We now enrich our results by taking into account the number of fixed points, thereby recover- 
ing Theorem 3. Recall from Proposition 6 that the label of the involution obtained by inserting 
n+ 1 as a fixed point in t e 3„ is (oi + l, 02 + 1, . . . , ae + 1) if m is odd, (1, 02 + I, . • ■ , 0£ + l) oth- 
erwise. Hence, if we keep track of the number of fixed points by a new variable s, the functional 
equation of Proposition 7 becomes: 

G(v; t, s) = vi + stviG(v; t, s)xm=i + stviG(v 2 , v 2 , v 3 ,..., vf, t, s)x m =o 

ft 

G(y;t,s) - G(vi, . . . , Vj-i, v j+ i, v j+ i, v j+2 , ■ . .,v e ;t,s) 



Vj - v j+1 

The series G(l, . . . , 1; t, s) counts involutions of !jK m ) by their length and number of fixed points. 
The change of variables (13) now gives 

K(x; t, s)G(x; t, s) = 1 + stG(0, x 2 , . . . , x& t, s)x m =o 

1 

~ t/,X$G(x\i . . . ,Xj-2,Xj-i + Xj,0,X J + i, . . . ,Xi]t,s), 
3 = 1 

where 

K(x; t, s) = 1 - t{x\ H Vxt) — stx m =i - t(xi H h xi), 

and 

G(x; t, s) = G(x!, ...,x e ;t) = G [ \ ■ r, \ ■ r, . . . , 1 ; t, s] . 

\l-t(x 1 -\ h x e ) l-t{x 2 -\ Vxi) 1-txg J 
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• If to is odd, the argument of Sections 4.2, 4.3, 4.4, applies verbatim. The only difference is 
that the term t occurring in the kernel is replaced by st. This gives at once the first part of 
Theorem 3, in the form (3). 

• If to is even, the equation reads: 

K(x\ t)G{x; t, s) = 1 + t(s — xi)G(0, X2, ■ ■ ■ , t, s) 

£ 

- ty~]xjG(xi, . . . ,Xj-2,Xj-i + Xj,0,Xj+i, . . . ,xe;t, s) 

with K(x; t) = 1 — t{x\ H — • + X£ + X\ + • • • + Xf)- We multiply it by 

M(x; s) := x 2 x\ ■ ■ ■ x l l ~ 1 (s — xi)---(s — x e ), 
and then argue as in Section 4.2 to conclude that 

2^ e{a) a(M(x; s)G(x; t, a)) = — — , (23) 

with the above values of K(x; t) and M(x; s). 

Now we cannot follow exactly the argument of Section 4.3, because <r(M(x; s)) does not differ 
from M (x; s) by a monomial. So it does not help to divide the equation by (s — x\) • • • (s — xi). 
Instead, let us leave the equation as it is, and extract all terms of the form X-y X^ ' ' ' X n with 
a > 0. More precisely, for a series F(xi, . . . , xi; t, s) in Q[x\, . . . , xi, s][[i\], let us denote 



r x >o x i 



t'^Ffa, ...,xt,t, s) := A \x\x\ ■ ■ ■ x\- 1 \F{x x , . . . , x e ; t, s). (24) 

a>0 

Consider the term 

a(M{x; s)G{x; t, s)) = a{x2x\ • • • x t e ~ 1 (s — x\) ■ ■ ■ (s — xi)G{x; t, s)) . 

Let us decouple in a the sign changes e±, . . . , eg, and the permutation tt of the x^s, as in (21). 
We wish to determine [xf °x\ • ■ ■ x 1 ^ 1 ] er(M(x; s)G(x; t, s)) . 

- If one of the ej's, for i > 2, is —1, then all monomials occurring in a(M(x; s)G(x; t, s)) 
involve a negative exponent and thus do not contribute. 

- If e\ = —1 while e,; = 1 for i > 2, the only way to obtain a non-zero contribution of 
a (M(x; s)G(x; t, s)) is to take 7r = id, and the contribution is then 

s £ G(0, ...,0;t,s). 

- If a = 7r G &£, the contribution is 

(s - xi)[xy°xI ■ ■ ■ x^ 1 } ((s - x 2 ) ■ ■ ■ (s - xe) ir(xl ■ ■ ■ x\~ x G{x;t, s))) . 

We note that this is a multiple of (s — x\). 
Hence, the result of our coefficient extraction on (23) is 

-s f G(0, . . . , 0; t, s)+{s-xx) ^ e(w)[xf°xl ■ ■ ■ x^ 1 ] ((s - x 2 ) ■ ■ ■ (s - x e ) n(xj ■ ■ ■ x l f l G{x; t, s)j) 



>o i i-u T,aeB e e (°") <r(M(x; s)) 
i x 2 ' ' ' x e \ 



1 K(x;t) 
Let us specialize this to X\ = s: 



-s e G{0,...,0;t,s) 



>o i J2a £ B e £ (< 7 ) <?(M(x;s)) 

h X2 "' X * 1 K(x;t) 
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The kernel K(x; t) is independent of s. But this is also the case of 

e(a) a(M(x; s)) = £ e(n)n (f[((s - x^ 1 (s x^- 1 ) ) 

a£B e Tte6e \i=l ) 

= -ict ( • .«•;), . .. , 

as is seen by taking linear combinations of rows. We have thus obtained the following counterpart 
of Proposition 10. 

Proposition 11. If m = 2£, the ordinary generating function of involutions avoiding (m + 
l)m • • • 21, counted by the length and number of fixed points, is, with the notation (24): 

G m (t,xi) := J2 t^x{ iT) = - lr ~-°~ 1 ^ dCU ' r '' " ' 



X ^ X 2 " " ' X d 



1 - t(x\ H h X£ + xi H hs/) 

Equivalently, the exponential generating function of these involutions is 



Gt\t, Xl ):= £ ^x{ M =~l[x^^--xndet((^-4) e *^)) 



<i,j<t 



We can now perform the coefficient extraction explicitly in the expression of Gm(t,xi] 



ct\t, xi ) = -\ 2 ew[.n((-i (i) -^ (i v t( - +2i) )n[-r l K^ w -< (8) ) et( " + " ,) ) 

1 ttGSc i=2 
1 - 

— X! £ i 7T )^2 x l( I k+^(l) -4-7r(l))n(%7r(i)-l - -fi-TT(i)-l) 



1 7r<ES« A;>0 i=2 



fe>0 



(/fc-j - Ik+j)l<j<£ 
V-i+j-l ~ Ii-j-l)2<i<&,l<j<t 



Upon extracting the coefficient of x\ , this gives the second part of Theorem 3. 

5. Permutations with no long ascending subsequence 

We now want to derive from the functional equation of Proposition 5 the Bessel generating 
function of permutations avoiding 12 • • • m(m + 1), given in Theorem 1. We follow the same 
steps as in the case of involutions, but the coefficient extraction is more delicate. 

5.1. In VARIANCE properties of the kernel 

The kernel of the equation of Proposition 5, that is, the coefficient of F(v;t), reads 

m 

EV-j-l V-j 

3= 

Its invariance properties appear clearly if we set 



1 



Xl 

Indeed, the kernel then becomes 



K(x;t) :=l-t{x 1 + --- + x m ), (25) 
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with x% = 1/xi, and is invariant under the action of the symmetric group 6 m , seen as a group 
of transformations of Laurent polynomials in the Xi . This group is generated by to — 1 elements 
of order 2, denoted ai, . . . , <J m -\- 

(Xj(P(xi,. ■ -,X m )) = P{X1,. . . ,Xj-l,Xj + l,Xj,Xj + 2, ■ ■ ■ ,Xm)- 

The functional equation now reads 

m— 1 

K(x;t)F(x;t) = 1 - t ^ x j+ iF(xi, . . .,Xj-i,Xj + x j+1 ,0,x J+2 , . . .,x m ;t), 
i=i 

with 

F{x-t)=F{x 1 ,...,x m -t)=F[— , -J- ;t] . (26) 

\xi xi + x 2 xi + ---+x m J 

5.2. Orbit sum 

Let us multiply the equation by 

M(x)=x° 1 xl---x™-\ (27) 

This gives: 

K(x]t)M(x)F(x;t) = M(x) 

m—1 

t ^ ^ X-^ • • • Xj Xj_^_-^Xj_^_2 ' ' ' -^('^•l 7 • ■ • ; ; *Gj H~ + 1 •> 0; *^i+2 j * • * j j ^) • (^8) 

i=i 

The j** 1 term of the sum is invariant under the action of the generator <7j (which exchanges xj 
and Xj + i). Consequently, forming the signed sum of (28) over the symmetric group & m gives 
the following orbit sum: 

J2 e(a)a(K(x;t)M(x)F(x;t))= £ e(a)a(M(x)), 
o-ee m o-ee m 

or, given that K(x;t) is © m -invariant, 

/ x I i\/rf si?, t \\ E CT g 6m £ ( g ) <r(M(x)) det(x}- 1 )i< t , J < m 
J^j(a)a(M(x)F(x;t)) = — ) = _ , (29) 

where K(x;t) is given by (25) and M(x) by (27). 

5.3. Extraction of F(l,...,l:i) 

For 1 < j < m, let us now denote Zj = x± H + Equivalently, = — with zo = 0. 

All series occurring in the orbit sum (29) become series in t with coefficients in Q(zi, . . . , z m ). 
In particular, 

F(x;t) = F[— ;t 

has coefficients which are Laurent polynomials in the Zj's. This is not the case for all terms 
in (29). For instance, if a is the 2-cycle (1, 2), 

. - , NN / 1 1 1 \ / 1 1 1 
a (F (x;t))=F — -t) = F( ;f 

\x 2 x\ + x 2 xi H + x m J \z 2 - z\ z 2 z m 

involves coefficients which are not Laurent polynomials in the Zj's. In order to perform our 
extraction, we will expand all rational functions of the Zj's as (iterated) Laurent series, by 
expanding first in z\, then in z 2 , and so on. For instance, the expansion of 1/ (xi +X3 +X4) reads 



E (-^i) ei \p (&\ + e 2 \ (-zi) ei Z' 

( V.a — 7nVl + I " ^ 



xi+x-s + x^ za-z 2 + zi ^ (z 4 - z 2 ) ei+1 ,y. n \ e\ 

ei>0 ei>0,e 2 >0 x 



l+ei+e2 
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In other words, the coefficients of our series in t now lie in the ring of iterated Laurent series in 
zi, . . . , z m , which is defined inductively as follows: 

- if m = 1, it coincides with the ring of Laurent series 1 in z\ (with rational coefficients). 

- if m > 1, it is the ring of Laurent series in z\ whose coefficients are iterated Laurent 
series in z<i , . . . , z m . 

It follows from this definition that an iterated Laurent series in the Zj's only contains finitely 
many non-positive monomials, that is, monomials z^ 1 ■ ■ ■ z^ 1 with ej < for all j. This allows 
us to define below a linear operator A, which extracts from an iterated Laurent series some 
coefficients associated with non-positive monomials and adds them up. 

Definition 12. Let A be the linear operator defined on iterated Laurent series in z\, . . . , z m by 
the following action on monomials: 

ac ei e m \ _ / 1) if e-L <0,...,e m <0 and ej =0=> e j+1 = ■■■ = e m = Q; , , 

' ' ' Zm > ~ \ 0, otherwise. [6U) 

Remark. The action of A can also be described as the extraction of a constant term: for any 
iterated Laurent series F(zi, . . . , z m ), 



A(F(z l7 . . . , z m )) = [z° 1 ... z° m ] F( Zl , z m ) E II 



i=0 3=1 



J 



This operator has been designed to extract from (29) the series F(l, . . . , 1; t) in which we are 
interested. The following proposition is thus the counterpart of Propositions 9 and 10. 

Proposition 13. The ordinary generating function of permutations avoiding 12 • • • m(m + 1) is 
obtained by applying A to a rational function: 



det(ij 3 )i<i,j<m 
1 - t(xi H h x m 



F m (t):= <|r ' =A 

with Xj = zj — Zj—i and zq = 0. 

Equivalently, the exponential generating function of these permutations is 



° Jl<i,j<m 



Remarks 

1. The fact that the action of A can be described as a constant term extraction, combined with 
closure properties of D-finite series [28, 27], implies that the series F m (and Fm ) are D-finite. 
This was first proved by Gessel [16]. 

2. Again, the determinant is a Vandermonde determinant and can be evaluated in closed form, 
but this will not be needed. 

Proof. We will prove that for all a £ & m , 

( cr{M(x)F(x;t)) \ _ f F(l, . . . , 1; t), if a = id; 

^ M(x) J \ 0, otherwise, 1 ' 

so that the first result directly follows from (29), after dividing by M{x) and applying A. It is 
then easily converted into an expression for the exponential generating function. 

Recall the definition (26) of F(x; t), and that G m acts by permuting the Xj's. Also, 

F(v 1 ,...,v m ;t)= J2 «? a(T) " 1 «r (T) ~ Oa(T) ---^ l+1 - ™ (T) * lT| , 



^Recall that our Laurent series only involve finitely many negative exponents. 
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where the labels ai(r) are defined by (4). This definition implies that, if Oj(t) = a 3 -+i(r), then 
Oj(t) = Oj + i(t) = • • • = a m (r) = |t| + 1. In other words, the v-monomials occurring in F(v;t) 
satisfy a property that should be reminiscent of the definition of A: 

F(v u ...,v m ;t)= c(e 1 ,...,e m )v e 1 1 v^---v^+- +em , (32) 

(e lt ...,e m )e£ 

where 

£ = {(ei, . . . , e m ) G N m : e, = e J+1 = • • ■ = e m = 0}. 
With this property at hand, we can now address the proof of (31). If a = id, 
' a (M{x)F{x;t))' 



A 



M(x) 




by definition of A and (32). 

There remains to prove the second part of (31). Let us consider an example, say m = 5 and 
a = 13425. Let r G ©4 , an d denote e,; = ai + i(r) — a^r), with ai(r) = 1 and a„ l+ i(r) = |t| + 1. 
Of course, > for all i. Up to a factor t' T ^, the contribution of r in <r (M(x)F(x;i)) /Af(x) is 

1 / x 2 x\x\x\ 
x 2 x\x\x\ a (xi + x 2 ) e2 ■ ■ ■ (xi H h x 5 ) e5 

2 3 4 
X^X i X 2 X 5 

X 2 x\x\x\ xf (Xl + X 3 ) e2 {Xi +X 3 + Xi) e3 {Xi + X 2 + X 3 + x 4 ) e4 (xi H h x 5 Y 5 

{z 2 - zi) 2 



(z 3 - ^)(^4 - ^s)^ 1 (z 3 -Z 2 + Zl Y 2 (Z4 -Z 2 + ZiY^Z^Z^ ' 



To prepare the Laurent expansion in the variables Zj, we rewrite this fraction as 

(Z2 - z x f 



e2 



2l ei Z, 1+e2 z] +e3+e4 Z? 5 f 1 - 23 Wl - ^) 1+C2 (l - I 



It is now clear that, in each term of the Laurent expansion, z 2 has a non-negative exponent, 
while Zi has a negative exponent. By definition of A, this implies that 

A / 1 ^ / M(x) \\ = Q 

\M(x) ° \x^{x\ +x 2 Y 2 ■■■ (xi + ■■ ■ + x 5 ) e V/ 

As this holds for every permutation r £ &n \ we have proved that (31) holds for a = 13425. 

Let us say that a series of Q(xi, . . . , x m )[[t]] is positive in Zj (or Zj-positive) if, in every term 
of its zr-expansion, Zj appears with a positive exponent. We define similarly the notion of Zj- 
negative, -non-positive, z 3 -non- negative series. We have just observed that, for m = 5 and 
a = 13425, the series a (M(x)F(x;t)) /M(x) is non-negative in z 2 but negative in Z4. This is 
generalized by the following lemma. 

Lemma 14. Take a G & m \ {id}. Let o~(j) be the largest non-fixed left-to-right maximum of a. 
That is, 

for k < j,o~(k) < o~(j), and for every k such that o~(k) > o~(j), one has cr(fc) = k. 

Let o~(i) be any value that is not a left-to-right maximum and satisfies o~(i) < i. For e\ > 
0, . . . , e m > 0, consider the fraction 

1 ( MM \ m) 

M(x) \x e i{xi+x 2 Y 2 ---(xi + ---+x m Y m J' 
Then this fraction is non-negative in z a u\ but negative in z^uy Since o~(i) < o~(j), applying A 
to this fraction gives 0. 
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Returning to the example a = 13425 studied above, we observe that the lemma applies with 
a(i) = 2 and a(j) = 4. 

This lemma implies the second part of (31): indeed, the contribution of any r G @( m ) in 
a(M(x)F(x;t))/M(x) is of the form (33). Hence proving the lemma will conclude the proof of 
Proposition 13. I 



Proof of Lemma 14. We establish this lemma via a sequence of three elementary properties. 

Property 1. Let i\ < %i < ■ ■ ■ < ik, and e G Z. The fraction 

1 



(± Zn ± ■ ■ ■ ± z lk y 

is non-negative in Zj 15 . . . , Zi k _ 1 . If e > 0, it is non-positive in Zi k , and even negative in Zi k if 
e > 0. 

Proof. The result is obvious if e < 0, as the fraction is a polynomial in this case. If e > 0, we 
prove it by induction on k. It clearly holds for k = 1. If k > 1, we write 

1 1 



(± Zll ± ■ ■ ■ ± Zlk y 



= E 



n>0 



e-l + n^ (izij™ 
ri 



(±z i2 ±---±z ife ) e +« : 



and conclude by induction on k. 

Property 2. Let a, j , e%, . . . , e m 6e as m Lemma 14. TTie fraction 

1 



(Xl+X 2 ) £2 -- -(.Tl + •••+. T m ) £ 

is non-negative in all z a ^ such that cr(fc) is no£ a left-to-right maximum, and non-positive in 
Mi) ■ 

Proof. It suffices to prove that the result holds for each term 

= 7 " ( 34 ) 

, (xi H h ^) e « / (2 ct(1 ) - H h z CT (£) - z CT (f)_i) e « 

for £ G {1, ... , m} (with z = 0). 

By Property 1, this term is non-negative in all variables, except possibly in z max ([T(i), ...,a(£))- 
Since max(cr(l), . . . ,&(£)) is always a left-to-right maximum, this proves the first part of the 
property. 

Consider now the variable ^cr(j) ■ 

- If I < j, then max(cr(l), . . . , &(£)) < so that the term (34) is independent of z a u) : 
and thus non-positive in this variable. 

- If j < I < o~(j), then max(tr(l), . . . , u{£)) = o~(j). Then (34) is non-positive in z a tj\ by 
Property 1. 

- Finally, if I > a(j), then {a(l), . . . ,a(£)} = {1,...,^}, so that the term (34) simply 
reads l/z^ 1 . This is independent of z a (j\, and thus non-positive in this variable. 

■ 

Property 3. Let a and j be as in Lemma 14. The fraction 

a(M(x)) 
Mix) 

is non-negative in all z a tv\ such that cr(fc) < k, and negative in z^/jy 
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Proof. We have 

M(a;) -ll^W-MO-i) 

Assume er(fc) < /c. The two terms of the above product that involve z CT (fc) are (z^)— 2 <T (fe)-i) fe_ ' T ^ fc ' ) 
and (z<7(fe)+i — -Zcr(/c)) e , with e = a (a(k) + 1) — cr(fc) — 1. The former term is non-negative in 
z a(k) because o~(k) < k. The latter term is non-negative in z a nA by Property 1. This proves the 
first part of the property. 

The two terms that involve z a (i) are {z a {j) ~ z a(J)—iY~ ^ an< ^ ( z <?(j)+i ~ z <?(j)Y \ w itli e = 
a ~ { a (j) + 1) — ~ !• Since cr(j) > j, the former term is negative in 2 a (j) by Property 1. By 
construction of j, the exponent e is 0, so that the latter term is simply 1. B 

Lemma 14 now follows by combining Properties 2 and 3. B 
5.4. Determinantal expression of the series 

Let us write e tx = X)b>o(^) & /^'- ^he second formula in Proposition 13 reads: 

f&H hb 



TGS< m ) 1 1 6i,...,6 m >0 <r£6 m V w V 



where M(x) = X2X§ ■ • • a;™" 1 , & = (&i, • ■ ■ , b m ), and x b = ■ ■ ■ x^ 1 . We will give a closed form 
expression of A ^ m^I^J, ) ) (Lemma 17), which in turn will give a closed form expression of the 
sum over a occurring in (35). 

Proposition 15. For b = (h, b m ) <G N m , 

^ ( a{M{x)) \ (b 1 + --- + b m )l tj (h . , , 

a£S m V \ J \ J/ lLt=l\ 1 I l<i<j< m 

Let us delay for the moment the proof of this proposition, and derive from it Gessel's deter- 
minantal formula (Theorem 1). 

Proof of Theorem 1. The exponential generating function of permutations of g>( m ) now reads 

V — - V + TT (b ._ i _ b . + j) 

reel"") 1 1 6i,...,6 m >0 lll=l I l<i<j<m 

Replacing t by t 2 , and taking the Bessel generating function gives 

f2\r\ £2(hiH h&m) 

E p?= E nri6i!(foi _ i + m) , II (6.-i-6, (36) 

r ge( m ) bi,...,b m >0 11 J=1 4 v 1 ; l<i<j<m 

But this is exactly Gessel's determinant. Indeed: 

TCI 

det(7j_i)i< iii < m = ^2 £ ( a ) Il^f*)- 1 



e ^ni 



= E n m i w -i + my. dct ((bi ' i+J + 1) "' (tri + m)) ' 

and this coincides with (36), because the above determinant is the Vandermonde determinant in 
the variables ui := bi — i (since (pi — i + j + 1) • • • (&, — i + w) is a polynomial in of dominant 
term u™'~°). a 
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There remains to prove Proposition 15. The proof relies on two lemmas. The first one is a 
simple identity based on a partial fraction expansion. The second gives a closed form expression 

°fA(4^) ; for& e Pr. 

Lemma 16. Let x\, . . . , Xk, U\, . . . ,Uk be indeterminates, and let the symmetric group &k act 
on the Xi 's by permuting them (that is, r(xi) — x T u\ for r G 6k). Then 

Proof. Let us denote u = (u\, . . . , Uk) and 



T(x,u) = I 



(x z + Uj) ■ ■ ■ {Xj + u k ) 

Xi+m-\ Vxk+Uk 

2—1 

This is a rational function of Uk, in which the numerator and denominator have degree k — 1. 
By a partial fraction expansion, 

T(x, u) = C(x, u) + ]T °f X > U } , (37) 

^ x e + u e H hit+ut 

where C and the o^'s are independent of By letting Uk tend to infinity, one obtains 

C(x,u) = (Xi + Uj). 

l<i<j<k 

The value of ai is obtained by taking the residue of T(x, u) at Uk = — (xt + U£ + ■ ■ ■ + Xk)- This 
gives, for £ < k — 1: 

, s riKj^xfc^ + M j)rii< l <A ; (^ - (xe +u e + ■■ ■ +x k )) 

ai{x,u) = =i = -, TT - ■ 

\\i±l,i<k\ Xi + U 'H 1" x k-i + Uk-i + Xk — (xe + Uf -\ h Xk)) 

Return now to (37). It is easy to check that ae(x, u)/(xe + ui + ■ ■ ■ + Xk + Uk) is left unchanged 
by the exchange of xe and xe+i- Consequently, the sum of the lemma reads 

e(r)r(T(x,uj)= £ e(r) r (C(x,u)) = det (f[( Xj + u h )\ = II ( x i~ x j)> 

rGSfc TGSfc \/i=i / i<i,j<k 1 < i <j<k 

because H^=i ( x j + u h) ' 1S a polynomial in Xj of leading term x k ~ l : the sum over r thus reduces 
to a Vandermonde determinant. ■ 

Lemma 17. Le£ b = (b 1} . . . , 6 m ) G N m and cr G 6 m . Let 

1 o-(M(ar)) 



x e M(x)a(x b ) 7 

where as before M[x) = x<z ■ ■ ■ x™~~ . That is, e = (e\, . . . , e m ) where ej = b T u\ — r(i) + i and 
t = cr -1 . Let k = max{i : bi > 0} (if b = (0, ... ,0), we take k = 0). Then 



k , 

'e 4 + • • • + e fc - 1 



\^)=\ II ^ ' ' e .'_! J' ifa(j)=jforallj>k, (3g) 

0, otherwise . 

Remark. If a(j) = j for all j > k, and i < k, then + ■ ■ ■ + > 1. Indeed, if e 4 - + ■ ■ ■ + e k = 
b r (i) + • • • + fe T (fe) + (i + • • • + k) — (t(i) + • • • + r(k)) were non-positive, this would mean that 
{r(i), . . . , r(k)} = {«,...,&} and 6j = • • ■ = bk = 0, which contradicts the definition of k. 
However, e, may be non-positive, and in this case the above expression vanishes. However, 
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e; + k — i > 0. When we apply this lemma to prove Proposition 15, we will write the above 
product of binomial coefficients in the following equivalent form: 

(ei + ••• + e fc )! yj e t (e, + l)---(ej + fc-z) 
Ili<*<fc(ci + *-*)! tk «* + ••• +e fc 

Proof. For an iterated Laurent series inzi, . . . , z k , of the form R(z) = Yl n eZ k c ( n ii • • • j 1 ' ' ' z k k j 
we define the negative part of -R(z) by 

[z<]i?(z) = [z<---z<] J R(z):= ^ c(m,...,n fc ). 

ni<0,...,n fe <0 

Let us first prove that if / = (/i, . . . , /fe) £ Z fe , 



We adopt the standard convention that (?) = unless < b < a. Given that xi = Zi — z%—\, we 
have 

1 1 



J f r / \ ? 2 f \ f> 



1 — — 



If /j < for some i, the z-expansion of \jx^ only involves non- negative powers of z,, so that the 
negative part of 1/a;-' is zero. The right-hand side of (40) is zero as well, and thus (40) holds. 
Assume now /j > for all i. The proof works by induction on k. If k = 1 and fx > 0, 

For fc > 2, 

1 



n + j2 - l x 



E 



/ V /a-l /^ +A («3-*2K*-"(%-%-i)A 
n + h ~ l\ r < < n 1 



n> 

/i-l 

E 

n=0 

gCi*r I )C + *-:i-i A - , )n(* + -ri A - 1 



/a - 1 / 2 4 +/2 (*s -z*)t» — (z k - z fc _!)A 



by the induction hypothesis. Now 

/i-i 

E 



n=0 



n + / a - 1 \(n + h + ■ ■ ■ + h - 1 
/a-l A n + f 2 -l 
fi-t 

E 



'n + h + ■ ■ ■ + fk - 1\ (h + ■ ■ ■ + fk - 1 

V /a-l 

h + h + ■ ■ ■ + fk - lV/a + • • • + /fc - 1 
/i - 1 A /a - 1 

The last equality results from the classical binomial identity 

a 

E 

n=0 



("D 




r 












a 
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This gives (40). 

Let us now prove (38). Assume that a(j) = j for all j > k. This implies that ek+i = ■ ■ ■ = 
&m = 0. As argued just after the statement of the lemma, efc > 0. But then l/x e is negative in 
Zk, and involves none of the variables Zk+i, . . . , z m . Thus, by definition of A, 

(by (40)), and this gives the first part of (38). 

Assume now that there exists j > k such that o~(j) ^ j. Then there also exists j > k such 
that a(j) < j. Let us choose such a j. Then there also exists £ > a(j) such that t(£) < £. We 
have 

e "U) = b J ~ 3 + = ~3 + a U) < °> 
et = K{i) - t{£) + £ > 0, 

with £ > a(j). Let £' = max{p > a(j) : e p > 0} (this set is non-empty as it contains £). Then 
e^+i = • • • = e m = 0, and l/x e is non-negative in z a u\ but negative in zt>. By definition of A, 
this implies that A(l/x e ) = 0. H 

Proof of Proposition 15. Let us denote by SUM(fe) the sum we want to evaluate. Let k = max{i : 
hi > 0}. By Lemma 17, the sum can be reduced to permutations a that fix all points larger than 

k, and then we use the closed form expression (39) of A^ jjj^^b)) ■ This gives: 



SUM(6) = £ ( CT ) A 



<ree fc 



ojM(x)) 
M{x)a(x b ) 



E ( r ) ( & H + b k )l j-j (b T (i) - t(i) + i) ■ ■ ■ (6 r (i) - r(i) + fc) 



reSt nti(^(0 - r(i) + k)\ i\ b T{l ) ' r(i) +»+■•■ + 6 T(fe ) - r(fc) + fc 

(fei + --- + fefc)' rr ., . , , .s 
= — TTTi 11 (k -*-&*+ j)- 

n*=i(6* - *+*)' i<i<i< fe 

The last equality is the case x$ = bi — i, Ui = i of Lemma 16. It is easy to check that, given that 
b^ + i = ■ ■ ■ = b m = 0, the above expression coincides with 

(&! + ■■■ + b m y. jj 

LU-libi - 1 + m )- ,<rt- * 

as stated in Proposition 15. I 

6. Final comments 

Clearly, our proof of Theorem 1, dealing with permutations of &( m \ is more complicated than 
our proof of Theorem 2, dealing with involutions of j( m \ We still wonder if there exists another 
change of variables, another coefficient extraction or another way to perform this extraction 
effectively that would simplify Sections 5.3 and 5.4. 

Our approach is robust enough to be adapted to other enumeration problems. Consider for 
instance the set ©( m ) of permutations n of g>( m ) in which the values 1, 2, . . . , m occur in this 
order. That is, 7r _1 (l) < • • • < 7r _1 (m). Garsia and Goupil found recently a simple formula for 
the number of such permutations of (small) length n: if n < 2m, this number is [15, Coro. 6.2] 

n — rn / \ . 

i«i-»=gw("r)(^' <«> 

This was then reproved by Panova [32]. See also [1]. 

In terms of the generating tree described in Section 2.1, this means that one is only counting 
the nodes of the subtree rooted at the permutation 12 ■ ■ • m. The only change in the functional 
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equation of Proposition 5 is thus the initial condition: instead of 1 (which accounts for the empty 
permutation), it is now V\ ■ ■ ■ v m t m . Sections 5.1 to 5.3 translate verbatim, and we reach the 
following counterpart of Proposition 13. 

Proposition 18. The ordinary generating function of permutations that avoid 12 • • • m(m + 1) 
and in which the values 1, . . . , m occur in this order is obtained by applying the operator A of 
Definition 12 to a rational function: 

/ jm 

r J m) <lr ' = A {x i 2 ---x^- i (i-t(x 1 + ---+x m )) J-J m e{a)a \KU(^ + • 

with Xj = Zj — Zj-i and z = 0. 

Equivalently, the exponential generating function of these permutations is 

f.\r\ ( £m e t(2H \-x m ) / x \ „m-l 



We have not further pursued in this direction, but one could try to obtain a more explicit 
formula giving the whole generating function (which would imply (41) when n < 2m). 

As discussed at the beginning of Section 2.2, the generating tree for 12 • ■ ■ m(m + l)-avoiding 
involutions can be described using m catalytic variables. Since these involutions are equinu- 
merous with (m + l)m • • • 21-avoiding involutions, it is natural to ask whether one could derive 
Theorem 2 from this tree and the corresponding functional equation. This could allow us to 
address the enumeration of 12 • • ■m(m + l)-avoiding fixed point free involutions, for which de- 
terminantal formulae exist (obtained by applying Gessel's 8 operator [16] to identities (5.41) 
and (5.42) of [4], which are equivalent to Theorem 2.3(3) in [31]; see also Stanley's survey [39, 
Thm. 8]). The recursive construction we have used for involutions does not allow us to address 
this problem. 



Acknowledgements. The author thanks Aaron Jaggard for communicating an early version 
of his paper with Joseph Marincel [21]. 
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